This invention relates generally to storage systems, and in particular to the long term reliable storage of verifiable data in such systems.
Large organizations throughout the world now are involved in millions of transactions which include enormous amounts of text, video, graphical and audio information which is categorized, stored, accessed and transferred every day. The volume of such information continues to grow rapidly. One technique for managing such massive amounts of information is the use of storage systems. Conventional storage systems can include large numbers of disk drives operating under various control mechanisms to record, back up and enable reproduction of this enormous amount of data. This rapidly growing amount of data requires most companies to manage the data carefully with their information technology systems.
Recently, new standards have been promulgated from various governmental entities which require corporations and other entities to maintain data in a reliable manner for specified periods. Such regulations, for example the Sarbanes-Oxley Act and the SEC regulations, require public companies to preserve certain business information which can amount to hundreds of terabytes. As a result, such organizations seek technologies for the management of data in a cost-effective manner by which data which is accessed infrequently is migrated to low performance or less expensive storage systems. This factor combined with the continuing reductions in manufacturing costs for hard disk drives has resulted in disk drives replacing many magnetic tape and optical disk library functions to provide archival storage. As the cost per bit of data stored in hard disk drives continues to drop, such systems will be increasingly used for archival storage.
Traditional high performance disk-based storage systems for enterprise information technology are usually equipped with high performance, high reliability hard disk drives. These systems are coupled to servers or other computers using high speed interfaces such as FibreChannel or SCSI, both of which are known standard protocols for information transfer. On the other hand, personal computers and inexpensive servers often utilize low performance, lower reliability disk drives with conventional low speed interfaces such as ATA or IDE. The lower reliability and performance of such hard disk drives allow them to be manufactured in mass production with low prices. These low priced disk drives can often be used in storage system products for archival storage. Examples include the Clarion and Centera products from EMC, NearStore from Network Appliance, and BladeStore from StorageTek.
In archival storage, the archived data is accessed only intermittently, for example on the order of a few times per year. As a result, performance is not an issue in the usual situation, but reliability is still of utmost concern. In addition to the usual internal desires for the retention of information in a reliable manner, often the data on these storage systems is covered by governmental regulations which require that it not be lost or modified. In addition, the low frequency of access to the data allows system designers to design the system in a manner by which the disk drives are turned off when they are not accessed, thereby reducing power consumption. Unfortunately, keeping hard disk drives off for long periods of time can also cause corruption of the recording media and the read/write devices. In many such archival systems, intentional, or even accidental modification of the data, for example by manual operator intervention or by software, is blocked using secure authentication mechanisms. To maintain highest data reliability, any data corruption or sector failure on one of the hard disk drives needs to be recovered, or at least detected and reported.
Accordingly, there is a need for storage systems using disk arrays by which low reliability hard disk drives can be employed in a reliable way, yet be prevented from data corruption or loss of data.